12. Demo: Normalization

Part 1

Understanding Data Normalization in Python Part 1

This demo further explores the concept of data normalization, an essential step in preparing data for machine learning processes. The focus is on two methods:

• Non-Rolling Z-Score Normalization: A traditional method which normalizes features based on the dataset's overall z-score.
• Rolling Z-Score Normalization: Allows normalization over a rolling window, adding flexibility in dealing with time-series data.

Steps Covered

1. Data Preparation:

   • Understanding the dataset structure through exploration and visualization.
   • Utilizing state space representation for data insights.

2. Normalization Process Initiation:

   • Initialization of empty data structures to track and store normalized data.
   • Iteration through data columns excluding non-numeric (e.g., date columns).

3. Applying Scikit-Learn's Standard Scaler:

   • Utilizing Scikit-Learn for defining, fitting, and transforming data with z-score normalization.
   • Ensuring Pandas data compatibility with dimensions transformation.

4. Tracking Normalization:

   • Maintaining a record of fitted normalizers for future transformations back to actual values.

This process lays groundwork for understanding trade value conversions in future tutorials.

Part 2

Understanding Data Normalization in Python Part 2

Key Concepts

• Data Plotting: After normalizing data, visualizing it helps confirm that the transformation is successful. Values usually fall within a specific range once normalized.
• Static Normalization: Involves adjusting the entire dataset so the data fits within a particular scale (-1.5 to 2 in this example). Relationships between data points may alter slightly due to scaling.
• Rolling Normalization: A process applied to subsets of data rather than the entire dataset at once. It uses a rolling window to iterate through data points, allowing dynamic adjustment.

Steps for Rolling Normalization

1. Initialization: Begin by copying the static normalization setup.
2. Column Setup: Pre-define the data columns to ensure all desired features are included.
3. Iteration: Use a rolling window of a specified size (e.g., 20 rows) to apply normalization, continuously tracking progress through data rows.

Considerations

• Step Size: Choosing an appropriate step size is critical—experiment with different values to find the optimal outcome.
• Non-Numeric Data: Exclude non-numeric columns (like dates) from normalization.

This approach helps manage time-series data and is foundational for advanced data analysis techniques.

Part 3

Understanding Data Normalization in Python Part 3

Key Topics Covered:

1. Special Cases Handling:

   • First Window: When n=0, manage initial data differently.
   • Final Window: Occurs when the current row plus step size exceeds dataset length.

2. Logic Definition:

   • Normalize Before Loops: Set the standard scaler outside the loop to ensure efficient computation.
   • Column Management: Use pandas.loc to extract and manage data slices effectively.

3. Data Transformation:

   • Convert pandas data to numpy using .values.
   • Adjust data dimensions with reshaping for normalization compatibility.

4. Middle Windows Implementation:

   • Define logic for windows not included in the first and last special cases.

Additional Insights

• Optimize iterations with step size increments.
• Ensure smooth execution through the use of while loops to traverse all dataset rows.

This structured approach is pivotal for rolling z-score normalization on datasets.

Part 4

Understanding Data Normalization in Python Part 4

Key Concepts:

• Static Normalization retains data trends but can bias models, as trends might not persist over time.
• Rolling Normalization avoids bias by not preserving trends, crucial for accurate market predictions.

Data Visualization:

• Plotting data helps visualize normalization impacts.
• Static normalization could mislead models, unlike rolling normalization that presents less biased relationships.

Benefits of Rolling Normalization:

• Offers clearer data representation for models without historical trend bias.
• Focuses on understanding interactions among values rather than static trends.

Process Recap:

• Begin with a state-space matrix, preprocess by cleaning and feature addition.
• Normalize data through static methods and rolling methods.
• Compare outputs to understand the practical advantages for machine learning models.

Rolling normalization, by reducing reliance on static trends trends, presents a cleaner dataset ideal for model training and future data analysis.
Exploration of the normalized data confirms the effectiveness of rolling methods in reducing unhelpful biases.